DOCUMENT RESUME 



ED 400 313 


TM 025 688 


AUTHOR 


Reckase, Mark D. 


TITLE 


The Design and Field Test of the ACT Portfolio 
System. 


PUB DATE 


Apr 96 


NOTE 


14p.; Paper presented at the Annual Meeting of the 
National Council on Measurement in Education (New 
York, NY, April 8-12, 1996). 


PUB TYPE 


Reports - Evaluative/Feasibility (142) — 
Speeches/Conference Papers (150) 


EDRS PRICE 


MF01/PC01 Plus Postage. 


DESCRIPTORS 


^College Entrance Examinations; Cooperation; ,c Field 
Tests; High Schools; Language Arts; Mathematics 
Education; ^Portfolio Assessment; Portfolios 
(Background Materials); Science Education; ^Scores; 
Self Evaluation (individuals); *Test Construction; 
Test Reliability; Test Results; Test Validity 


IDENTIFIERS 


*ACT Assessment; American College Testing Program 



ABSTRACT 

The American College Testing Program (ACT) is field 
testing a portfolio assessment model. The field test is designed to 
determine whether it is possible to implement a portfolio assessment 
model on a national level that will result in scores that are of 
sufficient reliability and validity that they can be used for 
decisions at the student level. An early decision by the ACT was to 
develop the assessment with the direct collaboration of teachers. 
Representatives of national education organizations nominated schools 
to participate, and seven schools were selected as design partners 
for the development effort. Each appointed a teacher to the design 
team, which met to decide on a definition of the portfolio system and 
a concept for the field test. Eleven to 13 work sample descriptions 
were then developed for each of the content areas of language arts, 
mathematics, and science. At the beginning of the school year, the 
school determines five work sample descriptions that will be used for 
each course and students are given guidelines to help them in the 
formation of their portfolios, which eventually will include an item 
for each of the work sample areas. The five entries and the student’s 
self-reflective letter are sent to the ACT for scoring. Pilot 
implementation and study with 35 teachers resulted in a revised 
system that 22 schools will test in the 1995-96 school year. 
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The use of portfolios of students’ work to reveal the skills and knowledge that 
they have acquired through their educational experience is not a new concepts. 
Kneeshaw (1992) reports that as early as 1977 the Ministry of Education for the 
Province of Ontario encouraged teachers to establish "writing folders" to support 
instruction and evaluation. Since that time, portfolios have been used for various 
assessment activities with mixed results (Condon & Hamp-Lyons, 1994; Elbow & 
Belanoff, 1986; Koretz, McCaffrey, Klein, Bell, & Stecher, 1992). Callahan (1995) 
presents a very thorough summary of these applications and the issues involved in 
sound portfolio use so they will not be repeated here. Suffice it to say that portfolio 
applications to assessment have varied in their success, and there is not a single 
model for portfolio assessment. This history of portfolio use raises the question of 
whether it is possible to successfully implement an assessment model based on 
portfolios on a large scale. If the answer is yes, what are the characteristics of the 
successful portfolio assessment model? 

The purpose of this paper is to describe a portfolio assessment model that is 
being field tested by ACT. The field test has the goal of determining whether it is 
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possible to implement a portfolio assessment model on a national level that will result 
in scores that are of sufficient reliability and validity that they can be used for 
decisions at the student level. The process used for designing the model and the 
specific design features of the portfolio assessment system that resulted are presented 
here. Other papers in this session (Miller & Hsu, 1996; Welch, 1996; Wolfe, 1996) 
summarize some of the results that have been obtained from a pilot test of this system 
during the 1994/95 academic year. A fairly elaborate field test of this system is 
occurring during the 1995/96 academic year. The details of both of these studies will 
be presented following the description of the system components. 

The Design Process 

Design Partners 

Because the use of portfolios has been found to be time consuming (Fontana, 
1995) and to have its genesis as an instructional tool, an early decision in the ACT 
Portfolio Project was to develop the design for the system with the direct collaboration 
of teachers. Once that decision had been made, the question arose as to how the 
teachers that would be part of the project would be identified. Since one of the goals 
of the project was to develop a portfolio model that could be used throughout the 
United States, teachers that were involved in the development process should be from 
a cross section of schools from the country. Enough teachers should be involved so 
that issues that were critical to the design of the system would be identified and 



2 




3 



addressed, but a relatively small group was desired so that a strong working 
relationship would develop. 

The process for selecting the teachers was to ask representatives of a number 
of national education organizations to nominate schools to participate in the project 
that had an interest in innovation and that had a history of providing teachers with the 
time for staff development and other enrichment activities. A letter and application 
form were sent to the principals of the nominated schools. From those that returned 
the rather lengthy application form, seven were selected to be Design Partners for the 
ACT Portfolio Project. Table 1 lists the Design Partner Schools that were selected. 



Insert Table 1 about here 



Each of the Design Partner Schools agreed to appoint a teacher to the Design 
Team, to pay for substitute time for the teacher, and to be part of the project for three 
years. Project membership required that the schools pilot and field test the portfolio 
system at their school. Students would be asked to produce portfolios of their work 
and submit them to ACT for evaluation. The schools agreed to these arrangements 
before the portfolio system had been designed. They either had a high level of 
confidence in ACT or had principals that were risk takers, or both. 

In selecting the Design Partner Schools, the goal was to include schools that 
represented different types of communities, different parts of the country, and different 
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educational systems. After having worked with the schools for over two years, we 
believe these goals had been achieved. 

Design Components 

Work on the design of the ACT Portfolio system began during the Summer of 

1994. At that time, the Design Partners and three portfolio consultants met with the 

ACT Development Team to define the basic parameters of the System. Three tasks 

were accomplished at the initial design meetings. First, the Design Partners, 

consultants and ACT staff agreed on a definition of portfolio that would be used as the 

basis for the design. After reviewing many different definitions that were available in 

the literature on portfolios, the definition proposed by Meyer, Schuman & Angello 

(1990) was selected for use. Their definition is presented below: 

A portfolio is a purposeful collection of student work that tells the story of 
the student’s efforts, progress, or achievement in given areas. This 
collection must include 

(1) student participation in selection of portfolio content; 

(2) the guidelines for selection; 

(3) the criteria for judging merit; 

(4) and evidence for student self-reflection. 

The Design Team particularly liked that the definition called for a "purposeful" 
collection and the "story" of progress. Other critical features were "student 
participation in selection" and "student self- ref lection." 

In addition to the requirements of the definition, the Design Team also agreed 
on the following design assumptions: 

1. The contents of the portfolios should provide evidence that students have 
acquired skills related to national curriculum standards. 



2. Portfolio development should be embedded within classroom instruction so that 
it represents actual student work and so that it did not require substantial extra 
efforts by the teachers and the students. 

3. The portfolio system should be flexible so that it can be used in a variety of 
educational settings. 

4. The portfolio system should have sufficient structure that reliable evaluations of 
student work can be obtained. 

5. Initially, the portfolio system will focus on language arts, mathematics, and 
science. 

There was also an expectation that the portfolio system would evolve over time 
as the teachers and schools gained experience with initial system design. The Design 
Team believed that what they were attempting was sufficiently new that initial design 
concepts would likely need to be modified. Revision should be accepted as a natural 
process of design rather than as a problem to be avoided. 

System Design 

After agreeing on assumptions and a definition, the Design Team worked to 
develop a concept that would meet all of the requirements. The balance between 
structure and flexibility was particularly challenging to achieve. The ultimate result, 
which was tried out in the first pilot test, was to develop a menu of portfolio 
components called "Work Sample Descriptions" from which each school, and possibly 
each class, could select the structure for the portfolio. 
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A Work Sample Description is a general statement of an educational outcome. 

For example, the following is a Work Sample Description for mathematics: 

Choose a piece of work that demonstrates your ability to make 
connections between two (or among more than two) branches of 
mathematics (for example, an algebraic proof of a geometry theorem or 
a graphic solution to an algebraic problem). 

The Work Sample Descriptions are sufficiently general that they will apply to any 

mathematics course from grades 9 to 12 and a wide variety of student work will be 

responsive to the requirements of the Description. The Work Sample Descriptions are 

not prompts in that no specific assignment is given. Rather, the provide a framework 

for selecting work samples to demonstrate the student’s capabilities. 

Eleven to 13 Work Sample Descriptions have been developed for each of the 

content areas of language arts, mathematics, and science. Some of the Work Sample 

Descriptions cross the usual content boundaries -- for example, mathematics applied 

to science material -- as well as more traditional content based tasks and skills. The 

Work Sample Descriptions were developed from a detailed analysis of the national 

curriculum standards and state curriculum frameworks so that they meet the 

requirement to assess the content of the curriculum standards. 

While the menu of Work Sample Descriptions was expected to provide the 

match to the curriculum standards and the flexibility requirement by allowing schools 

to design their own portfolios, there was also a need to provide enough structure to 

allow reliable student-level scores to be obtained from the portfolios. To meet that 

goal, the Portfolio System was designed based on selecting five Work Sample 

Descriptions from the menu. A theoretical analyses presented by Reckase (1995) 
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suggest that if five entries in a portfolio are separately scored, the composite score 
from the portfolio should have a reliability that is sufficiently high to support reporting 
at the student level. 

Given these design decisions, the Portfolio System functions in the following 
way. At the beginning of the academic year, the school determines the five Work 
Sample Descriptions that will be used for each course or combination of courses. 

This can be done by the teachers, by school committees composed of parents, 
teachers, and administrators, or by whatever other process the school chooses. Once 
the Work Sample Descriptions are selected for the course, the teacher reviews the 
lesson plans to determine which class activities match the Work Sample Descriptions. 
There should be at least two activities that match each description so that the students 
will have a selection of work to choose from for the portfolio. 

At the beginning of the school year, the students are given a handbook that 
includes the Work Sample Descriptions that will guide the formation of their portfolios. 
They also receive instructions about the final form of the portfolio and how the 
selection of materials will be scheduled. During the year, the students temporarily 
store materials in the portfolio folder. At the end of the course, or the academic year, 
the students select the final entries for the portfolio, one for each Work Sample 
Description, and write a self-reflective letter addressed to the reader of the portfolio 
that explains why each entry was selected and what it tells about his or her abilities. 

The five entries and letter are packaged along with demographic information 
and sent to ACT for scoring. Each Work Sample Description has a six-point scoring 
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rubric designed to match the requirements of that Work Sample. In the scoring 
process, each Work Sample is scored and the portfolio is given an additional holistic 
portfolio score using a four-point rubric. For the pilot and field tests of the Portfolio 
System, the five entry scores and holistic score are reported back to the school. The 
full portfolios are also returned to the school. 

Future reporting of the portfolio evaluations is expected to use scale scores that 
take into account the differing levels of difficult and dimensionality of the various Work 
Sample Descriptions. Initial work on the scaling is presented in Miller and Hsu (1996). 

Along with the basic structure for the Portfolio System, the Design Team 
recommended that in-service training be made available to help teachers implement 
the system and understand the scoring process. To meet these needs, two 
workshops have been developed to support the portfolio process. The first is called 
the Facilitator’s Workshop. Each school that uses the System is required to have a 
facilitator at their site. This person, usually a teacher that will be using the Portfolio 
System, participates in a two-and-a-half day workshop that not only covers all of the 
details of the System, but also provides materials to support a workshop at the 
facilitator’s school. The expectation is that the facilitator will return to their school and 
inform all teachers that will be using the System about the meaning of the Work 
Sample Description, and about the details of implementing the process. 

The second workshop presents the portfolio scoring process to one teacher in 
each curriculum area from each school. The purpose for this workshop is to have 
teachers become familiar with the criteria used to evaluate the portfolios and to 




8 



provide input into the scoring process. This workshop is not meant to train teachers to 
score the portfolios, but only to inform them about how they will be scored. 

The scoring of the portfolios is done through a rigorous process using highly 
trained individuals with expertise in the appropriate content area. All of the entries for 
a particular work sample are scored before the next one is started. Readers are 
trained prior to scoring each work sample, and scoring accuracy is carefully monitored. 

The Tryout Process 

Beginning in the Fall of 1994, the seven Design Partner schools began 
implementing the Portfolio System with teachers and students in each of the three 
content areas. Facilitators were trained for each of the schools, and materials for 
participating teachers and students were sent to the schools. Thirty-five teachers 
participated in the pilot study. Since this was the first implementation of a new design, 
the materials were necessarily less polished than they are now. Work Sample 
Descriptions were revised during the school year in response to recommendations 
from the teachers and consultants. The newness of the process and the frequent 
revisions caused some confusion, nonetheless several thousand portfolios were sent 
to ACT for scoring in the Spring of 1995. 

Information from the pilot implementation and the initial scoring was used to 
revise the system for a larger field test during the 1995/96 academic year. Twenty 
two schools are currently involved in the field test. During August, 1995 the facilitators 
for each of these schools attending workshops and these facilitators returned to their 




9 



schools to assist their teachers with implementation of the system. During February, 
1996, the scoring process workshop was held with 53 teachers from the 22 schools. 
This Spring, approximately 13,000 portfolios are expected to be shipped from the 
schools to ACT for scoring. The experience from the field test year and the scoring 
activities will be used to further refine the Portfolio System. 

Summary 

The purpose of this paper is to describe the initial development of the ACT 
Portfolio System and to present the current structure of the System. The ACT 
Portfolio is a system for collecting students’ class work in one or more of the areas of 
language arts, mathematics, and science. Structure is imposed on the portfolio by 
requiring that the portfolio entries conform to general criteria called Work Sample 
Descriptions. A specific course related portfolio is designed by selecting five Work 
Sample Descriptions from a menu of descriptions that has been developed to match 
the national curriculum standards. 

The portfolios also include a self-reflective letter from the student to the reader 
that indicates why the entries were selected and what they tell about the student. 

Each entry and the overall portfolio are scored according to rubrics that are specific to 
each Work Sample Description and each content area. 

The Portfolio System also includes staff development activities for school staff 
to cultivate local expertise in the portfolio system and to indicate the procedures used 
to score the portfolios. The System has gone through a pilot test and now is 
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undergoing a field test at 22 high schools. The System is being continually evaluated 
and revised. The goal is to produce a system that will meet the needs of the schools 
and that will provide reliable information about student capabilities. 
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Table 1 



Design Partner High Schools 



Du Sable High School -- Chicago, Illinois 



Cherry Creek High School -- Cherry Creek, Colorado 



Branford High School -- Branford, Connecticut 



Libertyville High School -- Libertyville, Illinois 



Culver City High School -- Culver City, California 



Mountlake Terrace High School -- Mountlake Terrace, WA 



Tupelo High School -- Tupelo, Mississippi 
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